<!DOCTYPE html>
<!-- saved from url=(0014)about:internet -->
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>
<meta http-equiv="x-ua-compatible" content="IE=9" >

<title>Chapter 6: Exercise 5</title>

<style type="text/css">
body, td {
   font-family: sans-serif;
   background-color: white;
   font-size: 12px;
   margin: 8px;
}

tt, code, pre {
   font-family: 'DejaVu Sans Mono', 'Droid Sans Mono', 'Lucida Console', Consolas, Monaco, monospace;
}

h1 { 
   font-size:2.2em; 
}

h2 { 
   font-size:1.8em; 
}

h3 { 
   font-size:1.4em; 
}

h4 { 
   font-size:1.0em; 
}

h5 { 
   font-size:0.9em; 
}

h6 { 
   font-size:0.8em; 
}

a:visited {
   color: rgb(50%, 0%, 50%);
}

pre {	
   margin-top: 0;
   max-width: 95%;
   border: 1px solid #ccc;
   white-space: pre-wrap;
}

pre code {
   display: block; padding: 0.5em;
}

code.r, code.cpp {
   background-color: #F8F8F8;
}

table, td, th {
  border: none;
}

blockquote {
   color:#666666;
   margin:0;
   padding-left: 1em;
   border-left: 0.5em #EEE solid;
}

hr {
   height: 0px;
   border-bottom: none;
   border-top-width: thin;
   border-top-style: dotted;
   border-top-color: #999999;
}

@media print {
   * { 
      background: transparent !important; 
      color: black !important; 
      filter:none !important; 
      -ms-filter: none !important; 
   }

   body { 
      font-size:12pt; 
      max-width:100%; 
   }
       
   a, a:visited { 
      text-decoration: underline; 
   }

   hr { 
      visibility: hidden;
      page-break-before: always;
   }

   pre, blockquote { 
      padding-right: 1em; 
      page-break-inside: avoid; 
   }

   tr, img { 
      page-break-inside: avoid; 
   }

   img { 
      max-width: 100% !important; 
   }

   @page :left { 
      margin: 15mm 20mm 15mm 10mm; 
   }
     
   @page :right { 
      margin: 15mm 10mm 15mm 20mm; 
   }

   p, h2, h3 { 
      orphans: 3; widows: 3; 
   }

   h2, h3 { 
      page-break-after: avoid; 
   }
}

</style>



<!-- MathJax scripts -->
<script type="text/javascript" src="https://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
</script>



</head>

<body>
<h1>Chapter 6: Exercise 5</h1>

<h3>a</h3>

<p>A general form of Ridge regression optimization looks like</p>

<p>Minimize: \( \sum\limits_{i=1}^n {(y_i - \hat{\beta}_0 - \sum\limits_{j=1}^p {\hat{\beta}_jx_j} )^2} + \lambda \sum\limits_{i=1}^p \hat{\beta}_i^2 \)</p>

<p>In this case, \( \hat{\beta}_0 = 0 \) and \( n = p = 2 \). So, the optimization looks like:</p>

<p>Minimize: \( (y_1 - \hat{\beta}_1x_{11} - \hat{\beta}_2x_{12})^2 + (y_2 - \hat{\beta}_1x_{21} - \hat{\beta}_2x_{22})^2 + \lambda (\hat{\beta}_1^2 + \hat{\beta}_2^2) \)</p>

<h3>b</h3>

<p>Now we are given that, \( x_{11} = x_{12} = x_1 \) and \( x_{21} = x_{22} = x_2 \). We take derivatives of above expression with respect to both \( \hat{\beta_1} \) and \( \hat{\beta_2} \) and setting them equal to zero find that,
\( \hat{\beta^*}_1 = \frac{x_1y_1 + x_2y_2 - \hat{\beta^*}_2(x_1^2 + x_2^2)}{\lambda + x_1^2 + x_2^2} \) and
\( \hat{\beta^*}_2 = \frac{x_1y_1 + x_2y_2 - \hat{\beta^*}_1(x_1^2 + x_2^2)}{\lambda + x_1^2 + x_2^2} \)</p>

<p>Symmetry in these expressions suggests that \( \hat{\beta^*}_1 = \hat{\beta^*}_2 \)</p>

<h3>c</h3>

<p>Like Ridge regression, </p>

<p>Minimize: \( (y_1 - \hat{\beta}_1x_{11} - \hat{\beta}_2x_{12})^2 + (y_2 - \hat{\beta}_1x_{21} - \hat{\beta}_2x_{22})^2 + \lambda (| \hat{\beta}_1 | + | \hat{\beta}_2 |) \)</p>

<h3>d</h3>

<p>Here is a geometric interpretation of the solutions for the equation in <em>c</em> above. We use the alternate form of Lasso constraints \( | \hat{\beta}_1 | + | \hat{\beta}_2 | < s \).</p>

<p>The Lasso constraint take the form \( | \hat{\beta}_1 | + | \hat{\beta}_2 | < s \), which when plotted take the familiar shape of a diamond centered at origin \( (0, 0) \). Next consider the squared optimization constraint \( (y_1 - \hat{\beta}_1x_{11} - \hat{\beta}_2x_{12})^2 + (y_2 - \hat{\beta}_1x_{21} - \hat{\beta}_2x_{22})^2 \). We use the facts \( x_{11} = x_{12} \), \( x_{21} = x_{22} \), \( x_{11} + x_{21} = 0 \), \( x_{12} + x_{22} = 0 \) and \( y_1 + y_2 = 0 \) to simplify it to </p>

<p>Minimize: \( 2.(y_1 - (\hat{\beta}_1 + \hat{\beta}_2)x_{11})^2 \).</p>

<p>This optimization problem has a simple solution: \( \hat{\beta}_1 + \hat{\beta}_2 = \frac{y_1}{x_{11}} \). This is a line parallel to the edge of Lasso-diamond \( \hat{\beta}_1 + \hat{\beta}_2 = s \). Now solutions to the original Lasso optimization problem are contours of the function \( (y_1 - (\hat{\beta}_1 + \hat{\beta}_2)x_{11})^2 \) that touch the Lasso-diamond \( \hat{\beta}_1 + \hat{\beta}_2 = s \). Finally, as \( \hat{\beta}_1 \) and \( \hat{\beta}_2 \) very along the line \( \hat{\beta}_1 + \hat{\beta}_2 = \frac{y_1}{x_{11}} \), these contours touch the Lasso-diamond edge \( \hat{\beta}_1 + \hat{\beta}_2 = s \) at different points. As a result, the entire edge \( \hat{\beta}_1 + \hat{\beta}_2 = s \) is a potential solution to the Lasso optimization problem!</p>

<p>Similar argument can be made for the opposite Lasso-diamond edge: \( \hat{\beta}_1 + \hat{\beta}_2 = -s \). </p>

<p>Thus, the Lasso problem does not have a unique solution. The general form of solution is given by two line segments:</p>

<p>\( \hat{\beta}_1 + \hat{\beta}_2 = s; \hat{\beta}_1 \geq 0; \hat{\beta}_2 \geq 0 \)
and
\( \hat{\beta}_1 + \hat{\beta}_2 = -s; \hat{\beta}_1 \leq 0; \hat{\beta}_2 \leq 0 \)</p>

</body>

</html>

